Updated: Nov 2nd, 2024
General Usage
Where do I store my files on Chronos?
How can I transfer files to and from Chronos?
Can I share files with others?
How can I check the current status of Chronos?
Can I close my computer and leave tasks running on Chronos?
Development
Using Python (JupyterLab Setup)
Can I use something other than Rstudio and/or JupyterLab?
Does Chronos support Dockers/Containers?
Misc
What if I have more questions or need help?
Chronos is our central lab server - primarily built to support large-scale computational work, including projects: (i) using Deep Learning/AI (ii) frequently handling large data-sets or which produce output files of significant size. Chronos currently has the following specifications:
OS: Ubuntu 22.04.5 LTS Kernel: 6.8.0-47-generic CPU: AMD Ryzen Threadripper PRO 5975 (32 Cores/64 Threads) GPU: NVIDIA RTX A6000 48GB RAM: 512GB* HDDs: 30TB (Total) Backup Power: Eaton 9E2000I UPS**
Chronos is optimized for multi-user workloads and we encourage you to freely use the system when needed, or as your daily work machine if you so wish. If Chronos is currently experiencing an extremely high workload (see: Section - Server Status), please consult with those currently logged in to know when resources will be freed. However, in our experience so far, there has been no need to do this.
Most importantly, Chronos was built with freedom in mind - the freedom to prototype and try new things without the fear of a monthly bill (i.e. this is not Google Cloud).
*Note: Chronos employs sophisticated memory compression techniques to significantly extend the capacity of the machine beyond 512GB of RAM. In practice, Chronos can sustain RAM usage up to ~800-1200GB, before overflowing onto disk-swap.
**Note: An uninterruptible power supply (UPS) device automatically provides backup power to Chronos during our frequent, but short-lived power cuts in the faculty, and prevents loss of work. ~20-100 minutes of power is provided depending on workload. A system-wide alert is issued, to any connected terminal, upon power loss in order to provide users an opportunity to save any and all work.
As required by the Technion’s security policy, you must be present in the faculty, on campus, or connected to the Technion VPN (VPN Setup) to access and use Chronos*.
Each user has their own, unique login credentials (username/password) and private working space within Chronos. For first-time users, your username along with a temporary password will be provided to you by an administrator.
Chronos is an OpenSSH server, meaning you can access it via any SSH-client using the following details:
Host: 192.114.18.118
Port: 22
Or directly on a SSH-enabled terminal using:
ssh username@192.114.18.118 #For example: ssh tim@192.114.18.118
For Windows users, we recommend MobaXTerm as an all-in-one SSH-client solution that will allow you to connect to Chronos but also manage/transfer files to and from the server (https://mobaxterm.mobatek.net/).
For R and Python users, access to a terminal is also provided directly through RStudio (see: Section: Using R - RStudio) and Jupyter Lab (see: Section: Using Python - JupyterLab) respectively - without the need to connect separately.
*Note: the initial VPN setup procedure can only be completed from within the Technion network (i.e. in the faculty, on campus).
To help keep Chronos secure, it is mandatory to change your temporary password after connecting for the first time. From any SSH-terminal (e.g. MobaXTerm, Rstudio, Jupyter Lab) use the following command:
passwd
And follow the instructions provided. If you forget your password, an administrator can reset it for you at anytime.
Each user on Chronos has a private storage area that nobody else can access or modify. This is located at:
/media/chronos/Storage/$USER #For example: /media/chronos/Storage/tim
For ease of use, a shortcut (‘UserSpace’) to this area is present in your /home/ directory (the default directory upon logging in). In other words, you can replace the above path with:
~/UserSpace
All user data is currently stored on a dedicated 20TB HDD. While we do not enforce a per-user storage limit, we do ask users to be mindful of others and refrain from storing duplicate files or unnecessarily large data on Chronos.
Files can be directly downloaded onto Chronos, or transferred from an external computer to your UserSpace (and vice versa), in a multitude of ways.
For Windows users, we recommend using MobaXTerm (https://mobaxterm.mobatek.net/) - an all-in-one SSH, X11-enabled client. MobaXTerm allows users to drag and drop files to and from your UserSpace, move or delete files, as well as open a remote web browser to directly download files onto Chronos*.
For example, after logging in to Chronos from MobaXTerm, use the following command to open a new browser session:
google-chrome & #Open a chrome instance, running on Chronos
During your first-run, please configure Chrome to automatically download files into your UserSpace, without prompt (i.e. disable “Ask where to save each file before downloading”). X11-forwarding does not fully support secondary windows, so a save dialog may not always appear.
Alternatively, files can be directly downloaded onto Chronos independently of a browser using a direct URL or through git. For example, from any connected terminal:
wget -P ~/UserSpace [URL] #For example: wget -P ~/UserSpace/Folder http://www.test.com/test.zip
git clone https://github.com/shenorrLabTRDF/TimeAx ~/UserSpace/Folder
Lastly, files can also be transferred or managed from within RStudio or JupyterLab.
Optimal transfer speeds will be achieved when connected to the Technion LAN (i.e. in the faculty/on campus) and will be considerably slower when connected to the VPN from home.
*Note: remote browser access requires the user to maintain an active connection to Chronos, which may not be suitable for extremely large downloads. Furthermore, this service requires X11 support (i.e. MobaXTerm) and will not work in all terminals.
Each UserSpace is incrementally backed up once a day, at 19:00, to our Dropbox account*.
Files stored outside of your UserSpace are not backed up. It is therefore mandatory Chronos policy that all work is stored in this area.
Dropbox limits us to low transfer rates of 10-12 files/second. Thus, in the rare cases where a UserSpace folder contains a significant number of files (especially small files, e.g. >50,000) etc., we recommend excluding this directory from backup using the DNBU (Do Not Backup) function and compressing the folder into a single file for backup instead. For example:
DNBU add ~/UserSpace/FolderToExcludeFromBackup #Exclude this folder and all contents from backup
To then compress the folder:
tar --use-compress-program='pigz -p 16' -cf ~/UserSpace/FolderToExcludeFromBackup.tar.gz ~/UserSpace/FolderFolderToExcludeFromBackup #Compress the folder, using 16 CPU cores
To remove an exclusion or check the current exclusion status of a given folder:
DNBU remove ~/UserSpace/FolderToExcludeFromBackup
DNBU check ~/UserSpace/FolderToExcludeFromBackup
*Note: Backups are mirrors i.e. if you delete a file on Chronos, it is deleted on Dropbox. However, Dropbox will retain a restorable copy of all deleted files for 6 months. Moreover, Dropbox will store versioning information for each file, allowing files to be reverted back to an older copy (as far back as 6 months) when needed. A future update to Chronos will enable a new interface that allows users to directly restore files from Dropbox without admin intervention.
Once connected, Chronos will automatically report its current status to you. For example:
---------------------- SERVER System: Ubuntu 22.04.5 Kernel: 6.8.0 Uptime: 6d|1h|2m ---------------------- MEMORY Memory (RAM): 29GB/251GB [===> ] 11% # = Compressed Memory ---------------------- DISK SPACE User Storage: 9TB/20TB [===============> ] 46% Env Storage: 1TB/10TB [> ] 3% OS Storage: 0.78TB/1TB [=========================> ] 78% SSD Health: 99% ---------------------- CPU Average CPU Usage (5m): 0% [> ] 0% CPU Temperature: 35.4°C ---------------------- GPU GPU Driver: 555.58.02 | CUDA Version: 12.5 GPU (VRAM): 248MB/49140MB [> ] 0% GPU Temperature: 32°C ---------------------- POWER Main Power: Online ---------------------- CURRENTLY ACTIVE: yonatan yochay merav tim ---------------------------------- CHRONOS GUIDE: Link (Ctrl + Click) ----------------------------------
While we recommend that users inspect this to gauge the current workload on Chronos, and time their own work accordingly, this is only rarely needed. If, for example, many resources are coloured in red - the server is under heavy load and you may need to consult with the users currently logged in (shown in the status report) to know when you can use Chronos if you intend to run a heavy task.
At anytime, you can view updated status using the following command on any terminal:
status
Or, alternatively, any user can follow processes in real-time using:
htop
Users work on Chronos in a multitude of ways, depending on the task at hand and the programming language being used. Sections specific to R, Python and Docker are found below.
An important feature of Chronos is the ability to leave a task running without the need to keep your connection open or laptop powered on. For RStudio users, this is managed automatically. For all other purposes, Chronos supports tmux (a terminal multiplexer), which allows the user to open any number of sub-terminals which remain active regardless of your connection to Chronos. For example:
tmux new -s ExampleSession #Create a new tmux session called 'ExampleSession'
From within this session, you can run tasks e.g. open a conda environment, initiate JuptyerLab and work in Python. The session will remain active, at all times, until closed by the user. To check your currently active sessions:
tmux ls
To exit a tmux session, without closing it, and re-enter it when needed:
tmux detach-client #From within an active tmux session to exit it
tmux a -t ExampleSession #From outside of a tmux session to re-enter a session
To permanently close a tmux session:
#From within an active tmux session:
exit
#From outside a tmux session:
tmux kill-session -t ExampleSession
Chronos runs RStudio Server, which provides a desktop-like, graphical interface for R on Chronos. To access RStudio from your own machine, simply direct your web browser to:
192.114.18.118:8787
And login with your Chronos credentials. As with everything, the normal connection rules apply i.e. you must be connected to the Technion LAN or VPN to access RStudio.
This is a user-specific R environment. All packages installed are for you and you only - as are all RStudio settings and processes being run. You are responsible for updating packages and are free to uninstall/install packages at will.
RStudio Server provides a ‘Terminal’ tab which allows you to interact directly with Chronos as you would from any SSH-terminal. This allows you to, for example, change your password or check the current status of Chronos (see: Section: Passwords and Section: Status) .
To transfer files to and from Chronos using RStudio, use the ‘Files’ tab which provides ‘Upload’ and ‘Export’ (download) functions (found in ‘More’).
Important: To free resources for active users, idle R sessions - sessions that have not executed code for >=3 hours - are written to disk and removed from RAM. When restoring your session (which RStudio will do automatically upon launch), please allow 1-10 minutes for it to transfer data from the disk back into RAM ready for use - the interface will not load until this has completed.
If RStudio fails to reload after a prolonged period of time - which can happen when a large amount of data was left inside your R environment - you can clear your saved session by connecting to Chronos outside of RStudio and issuing the following command on any terminal:
clearR #Clear and delete all data stored in the R environment
To avoid this and any loss of work, we recommend including code to save RObjects (e.g. .RDS files) throughout your scripts whenever running a large job and terminating your scripts with code that clears the REnvironment automatically:
rm(list = ls()) #Clear R environment
No R session currently running code will be considered idle. In other words, you’re free to close your browser/laptop and Chronos will continue to execute your script to completion - no matter how long it takes i.e. there is no need for tmux when using RStudio on Chronos.
To end a session, click the red ‘off’ button on the top-right of RStudio. This will also wipe and remove your current environment from both RAM and the disk.
The default working directory has been set to your UserSpace. If this is not the case, or the working directory was changed, please reset it:
setwd('/media/chronos/Storage/[username]') #For example: setwd('/media/chronos/Storage/tim')
All system packages typically required by R in order to compile packages are already provided by Chronos. Should you encounter additional needs, please contact an administrator.
Finally, common R packages used in the lab (including their dependencies e.g. ggplot2) have been pre-installed for each user:
CRAN/Github Packages:
devtools
remotes
Seurat
BiocManager
stringr
msigdbr
glmnet
stabs
ggpubr
BayesPrism
Bioconductor Packages:
SingleR
celldex
ClusterProfiler
tradeSeq
glmGamPoi
CATALYST
diffcyt
Note: R packages requiring access to Docker/Singularity e.g. dynverse are not currently supported on Chronos for security reasons.
Note: Installed packages are not backed up and cannot be restored in the event of removal. Please re-install packages instead.
Chronos supports Conda for the creation and management of self-contained Python environments. Conda is accessible from any Terminal connected to Chronos and all Conda installations are user-specific, and cannot be accessed, run or modified by others.
We assume Python users are already familiar with the use of Conda, however a starter guide is available here: Basic Guide
While a more advanced guide is available here: Detailed Guide
However, if you’re new to Python, please contact an administrator who can teach you all of the necessary details.
If you do not have Conda enabled for your user (i.e. the conda function is not available from a terminal) but you require it, please install it by following these steps:
#Change directory to your UserSpace:
cd ~/UserSpace
#Download the Conda installer:
curl -O https://repo.anaconda.com/archive/Anaconda3-2024.10-1-Linux-x86_64.sh
#Mark the installer as executable:
chmod u+x Anaconda3-2024.10-1-Linux-x86_64.sh
#Intiate Conda installation and setup:
./Anaconda3-2024.10-1-Linux-x86_64.sh
#Verify that Conda is now available to your user and delete the installer:
conda
rm Anaconda3-2024.10-1-Linux-x86_64.sh
And follow the instructions, making sure to enable (‘Y’) conda init at the end of the process. Conda, as a software package, will be installed into your home directory e.g. /home/tim/Anaconda3. To update conda to the latest version:
conda update conda
It is now mandatory that all Conda environments are stored separate from your UserSpace. Chronos provides a dedicated 10TB HDD for this purpose, located at:
/media/chronos/Env_Storage/$USER #For example: /media/chronos/Env_Storage/tim
To set this as your default Conda directory, use:
conda config --prepend envs_dirs /media/chronos/Env_Storage/$USER
To create a new Conda environment called ‘EnvName’:
conda create -n EnvName python=3.11 #Python 3.11 is used for example purposes only
conda activate EnvName
All Conda environments are available from within any tmux session and they do not need to be explicitly installed within one. To list all available Conda environments:
conda env list
Since Conda environments often contain ~200,000+ files, backup is not feasible. For that reason, Env_Storage is excluded from Dropbox. Instead we encourage users to export a configuration file for each environment to their UserSpace (which is backed up). For example:
conda activate EnvName
conda list --explicit > /media/chronos/Storage/$USER/EnvName.yml
Which serves as a backup, and allows for easy and rapid recreation of the environment:
conda create -n EnvName_Recreated --file /media/chronos/Storage/$USER/EnvName.yml
Finally, to delete a Conda environment:
conda remove --all -n EnvName
Chronos contains a high-end, server-grade GPU to power Deep Learning and other AI-related processes through Python. To facilitate this, all Conda environments have automatic access to the GPU.
To provide a multi-tool, graphical interface for Python, we recommend installing JupyterLab into each, individual conda environment. For example:
conda activate EnvName
conda install jupyter
We typically recommend using JupyterLab, as opposed to Jupyter Notebook, as it provides additional functionalities including a ‘Terminal’ tab which allows you to interact directly with Chronos as you would from any SSH-terminal.
To access Jupyter, the service must first be mapped to a user-specific port. To check which ports are available and select one to use, see here: Chronos Port Mapping
Add this port to your installation by first creating a Jupyter configuration file:
conda activate EnvName
jupyter lab --generate-config #Replace 'lab' with 'notebook' if using jupyter notebook
nano ~/.jupyter/jupyter_lab_config.py
Adding the following lines to the top of the file:
c = get_config()
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 5004 #Replace with your chosen port
And saving the file (ctrl + X, ‘Y’, ENTER).
For ease of use and in order to secure your Jupyter sessions, a password must be set using:
jupyter lab password
And following the instructions given.
JupyterLab can only access the files and folders stored within the directory it was launched from. For example, to launch JupyterLab with access to your entire UserSpace:
conda activate EnvName
cd ~/UserSpace
jupyter lab
To access Jupyter from your own machine, simply direct your web browser to:
192.114.18.118:[CHOSEN_PORT] #For example: 192.114.18.118:5004
Or use the desktop JuptyerLab application (https://github.com/jupyterlab/jupyterlab-desktop).
This configuration file, port and password will apply to Jupyter in all conda environments, however, if you wish to run multiple Jupyter instances simultaneously you will need to either generate separate configuration files (each with unique ports) for each instance or launch additional instances with a unique port directly:
jupyter lab --no-browser --port XXXX #For example: jupyter lab --no-browser --port 5005
Please ensure that you reserve any and all ports that you use in the linked spreadsheet above.
While RStudio and JupyterLab are the recommended way to interact with R and Python on Chronos, any Integrated Development Environment (IDE) that supports remote access through SSH is supported.
Among the many options available, we recommend using Visual Studio Code (https://code.visualstudio.com/), as alternative, with the official ‘Remote - SSH’ extension to run R, Python and manage files on Chronos.
Chronos supports a multitude of image container formats, including Dockers, however, for security reasons, Docker itself is not available. Rather, Chronos pulls, manages and runs images using AppTainer (formely known as ‘Singularity’). This provides a rootless and secure way for each user to run images.
To enable AppTainer for your user, please install it by following these steps:
#Create a folder to install Apptainer into:
mkdir /media/chronos/Env_Storage/$USER/Apptainer
#Download and install Apptainer:
curl -s https://raw.githubusercontent.com/apptainer/apptainer/main/tools/install-unprivileged.sh | \
bash -s - /media/chronos/Env_Storage/$USER/Apptainer
#Make apptainer available from anywhere on Chronos:
echo 'export PATH="/media/chronos/Env_Storage/$USER/Apptainer/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc
#Verify that Apptainer is now available to your user:
apptainer
To pull and run, as an example, a Docker image:
#Create a directory to store containers:
mkdir /media/chronos/Env_Storage/$USER/Containers
#Pull and run a docker image:
apptainer pull /media/chronos/Env_Storage/$USER/Containers/hello-world.sif docker://hello-world
apptainer run /media/chronos/Env_Storage/$USER/Containers/hello-world.sif
In the above example, AppTainer pulls the ‘Hello World’ Docker image, converts it to .sif image (the native container format of AppTainer) and runs it.
Beyond providing platforms for R/Python/containers, Chronos will also provide a suite of additional tools in the near-future:
Bavel not only provides a ChatGPT-like graphical interface accessible via any browser, but code-specific models will also be integrated into RStudio and JupyterLab to enhance troubleshooting efforts.
Bavel remains under active development and tutorials will soon be added to this guide.
Prometheus aims to provide users with a way to easily query databases (such as the upcoming IMMAGE/Ellison database) but also visualize, analyze and manipulate data in real-time all through an easy-to-use graphical interface accessible from any browser. Moreover, Prometheus allows users to build interactive dashboards to showcase their work to the team and to collaborators.
An example dashboard can be found here: Dashboard.
Prometheus remains under active development and tutorials will soon be added to this guide.
If you have additional questions, require support or encounter any problems please do not hesitate to contact an administrator (Tim, Yochay).